Generalizations of principal component analysis, optimization problems, and neural networks

-We derive and discuss various generalizations of neural PCA (Principal Component Analysis)-type learning algorithms containing nonlinearities using optimization-based approach. Standard PCA arises as an optimal solution to several different information representation problems. We justify that this is essentially due to the fact that the solution is based on the second-order statistics only. I f the respective optimization problems are generalized for nonquadratic criteria so that higher-order statistics are taken into account, their solutions will in general be different. The solutions define in a natural way several meaningful extensions of PCA and give a solid foundation for them. In this framework, we study more closely generalizations of the problems of variance maximization and mean-square error minimization. For these problems, we derive gradient-type neural learning algorithms both for symmetric and hierarchic PCA-type networks. As an important special case, the well-known Sanger's generalized Hebbian algorithm ( GHA ) is shown to emerge from natural optimization problems. Keywords--Principal components, Optimization, Neural network, Unsupervised learning, Nonlinearity, Robust statistics, Generalized Hebbian algorithm, Oja's rule. 1. I N T R O D U C T I O N Principal component analysis (PCA) is a well-known, widely used statistical technique. Essentially, the same basic technique is utilized in several areas under different names such as Karhunen-Loeve transform or expansion, Hotelling transform, and signal subspace or eigenstructure approach. In pattern recognition, PCA is used in various forms for optimal feature extraction and data compression (Devijver & Kittler, 1982). In image processing, PCA defines the Hotelling or KL transform that is optimal in image data compression (Jain, 1989). In signal processing, a useful characterization of signals is to assume that they roughly lie in the signal subspace defined by PCA. Several modern methods of signal modeling, spectrum estimation, and array processing are based on this concept (Therrien, 1992). Let x be an L-dimensionai data vector coming from some statistical distribution centralized to zero: E { x } = 0. The ith principal component x re ( i ) of x is defined by the normalized eigenvector c (i) of the data covariAcknowledgements: The authors are grateful to Prof. Erkki Oja for useful comments and insightful discussions on the topic of the paper, and to a reviewer for his detailed comments. Requests for preprints should be sent to Dr. Juha Karhunen, Helsinki University of Technology, Laboratory of Computer and Information Science, Rakentajanaukio 2 C, FIN-02150 Espoo, Finland. ance matrix C = E { xx r } associated with the ith largest eigenvalue h( i ) . The subspace spanned by the principal eigenvectors c( 1 ) . . . . . c (M) (M < L) is called the PCA subspace (of dimensionality M). PCA networks are neural realizations of PCA in which the weight vectors w (i) of the neurons or the weight matrix W = [w(1) . . . . . w(M)] converge to the principal eigenvectors c ( i ) or to the PCA subspace during the learning phase. It is well known that standard PCA emerges as the optimal solution to several different information representation problems. These include: 1. maximization of linearly transformed variances E{ [w(i)Tx] 2 } or outputs of a linear network under orthonormality constraints ( w r w = I ) , 2. minimization of the mean-square representation error E{ [Ix ill 2 }, when the input data x are approximated using a lower dimensional linear subspace = W W rx; 3. uncorrelatedness of outputs w (i) rx of different neurons after an orthonormal transform (WrW = I ) ; and 4. minimization of representation entropy. Derivations of the optimal PCA solutions with the required assumptions and constraint conditions can be found in several textbooks (see, e.g., Devijver & Kittier, 1982; Jain, 1989; Young & Calvert, 1974). The